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Abstract 


With the goal of codifying student learning outcomes of commu¬ 
nity-based research (CBR), the authors created a conceptually 
valid and statistically reliable CBR Student Learning Outcomes 
Survey. The project began with individual interviews and focus 
groups with 70 undergraduates and faculty at six colleges and 
universities nationwide discussing perceived benefits of CBR. 
Based on analyses of these interviews, five CBR outcome con¬ 
structs were derived: academic skills, educational experience, 
civic engagement, professional skills, and personal growth. The 
survey was piloted online in spring 2009 to students who had 
experienced CBR from 15 colleges and universities (N = 166). 
Factor analyses revealed strong statistical reliability across survey 
constructs. The authors invite faculty to use the instrument to 
assess CBR courses and invite students who have experienced 
CBR to complete the survey online through spring 2012, as part 
of a national study of CBR outcomes. 


Introduction 



s more colleges and universities have integrated experien¬ 


tial learning programs into their curricula, there has been 


an increase in research focused on identifying learning 
outcomes of such programs. Studies have identified a range of out¬ 
comes related to undergraduate students’ participation in service¬ 
learning and, to a lesser extent, community-based research (CBR), 
including increased engagement with academic studies, develop¬ 
ment of professional skills, and civic engagement. 

Although learning outcomes of service-learning and CBR 
are similar, CBR may have greater strengths in terms of academic 
engagement and deepening one’s understanding of one’s major, 
because identifying research questions and collecting data related 
to them develops and reinforces disciplinary knowledge in ways 
that service-learning may not. The overarching goal of the current 
research is to assess the effectiveness of CBR, begin to identify best 
practices, and examine the effects of various practices, based on 
diverse academic factors. 
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Several constituencies stand to benefit from the systematic 
assessment of CBR’s impact on students. Institutions of higher 
education will be able to show the extent to which CBR courses 
are contributing to their institutions’ missions and to students’ 

learning and career preparation. 
Faculty members will understand 
the impact of their CBR courses 
on student learning, and will be 
able to use data to improve their 
teaching, while also advocating 
for CBR as a rigorous pedagogy 
to colleagues. Students will be 
aware of the contribution of 
CBR to their learning experi¬ 
ence. Finally, community part¬ 
ners will better recognize how 
their participation in CBR pro¬ 
vides critical benefits to students 
(Gelmon, Holland, Driscoll, Spring, 
& Kerrigan, 2001). 

As a first step toward 
codifying the effectiveness of 
CBR, the authors developed an 
evidenced-based, conceptually 
reliable, and statistically valid survey instrument with the potential 
to quantify student learning outcomes of CBR classes. This article 
describes the development of the survey instrument. Constructs 
were based on extensive student and faculty interviews. The five 
scales that constitute the instrument reliably assess five commonly 
discussed dimensions of student learning related to service¬ 
learning and CBR: development of academic skills, enhanced 
educational experience, increased civic engagement, development 
of professional skills, and personal growth. 

Literature Review 

Outcomes of Service-Learning 

Throughout the research literature, proponents of service¬ 
learning express enthusiasm about the benefits to students at 
the college level (Coffey, 2010; Ghannam, 2007; Hart, 2006; Sherman 
& MacDonald, 2009). However, questions about the cognitive and 
affective benefits compared to direct instruction, a lack of clarity 


“Studies have 
identified a range 
of outcomes related 
to undergraduate 
students participation 
in...community- 
based research (CBR), 
including increased 
engagement with 
academic studies, 
development of 
professional skills, and 
civic engagement.” 
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about the politics and goals of service-learning, and the challenges 
of integrating service-learning experiences into the curriculum 
have led to caution in adopting this form of experiential learning 
(Eyler, 2000; Polanyi & Cockburn, 2003; Westheimer & Kahne, 2004). 

Advocates of experiential learning have urged researchers to 
document student learning outcomes through the use of multiple 
methodologies and presentation of solid evidence, in order to pro¬ 
vide a basis for replication and further research (Gelmon et al, 2001; 
Mehaffy, 2009). At the same time, researchers identify challenges 
in assessing service-learning outcomes (Keen, 2009; Marullo et al, 
2003; Pike, 2009). One challenge is that service-learning can take 
many different forms (e.g., voluntary or mandatory, integrated 
into coursework or not, involving reflection or not). In addition, 
service-learning can be studied at many different levels—including 
effects on students, faculty, community partners, and institutions 
themselves (Keen, 2009). Even when a specific outcome is identi¬ 
fied—for example, civic engagement—the concept can be defined 
very differently across different instruments, making it difficult to 
link studies that share similar outcome variables (Keen, 2009; Prentice 
& Robinson, 2007). Development of academic skills is often cited as a 
benefit of service-learning (e.g., David, 2009; Higher Education Research 
Institute, 2002; Vogelgesang & Astin, 2000), but what, exactly, consti¬ 
tutes “academic skills”? In the case of the student who says that 
service-learning has made her “more comfortable speaking up in 
class,” do the authors see this as development of an academic skill, 
social skill, professional skill, or personal growth? Becoming more 
comfortable sharing one’s perspectives in public settings could be 
an example of development in all four areas. This illustrates some 
of the challenge in codifying outcomes of CBR. 

Generalizing results of studies also can be difficult, since many 
published articles looking at student outcomes of service-learning 
do not distinguish between different delivery types (e.g., courses 
with a service-learning component versus courses dedicated to 
service-learning versus service-learning as a cocurricular activity). 
For example, studies have shown that outcomes of service-learning 
are enhanced when the service-learning includes a reflection 
component, or when faculty integrate the service-learning 
experience into class discussion, but whether such components 
were part of students’ experience is not always assessed (Conway, 
Amel, & Gerwein, 2009; Hunter & Brisbin, 2000). Therefore, while 
studies of student learning outcomes often focus on students’ 
self-reported changes on variables such as academic skills, civic 
engagement, and professional skills, such studies often raise 
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questions about the mechanisms by which students participating 
in service-learning experienced these benefits (Gelmon et al, 2001; 
Higher Education Research Institute, 2002). 

Although the reported benefits of service-learning are 
compelling, most studies of service-learning outcomes are not 
conducted with control populations. It is not always clear whether 
the benefits of service-learning outweigh the effort of implementing 
it, or what curricular trade-offs result, if any. In studies with control 
groups or that compare service-learning with non-service-learning 
alternatives, results are often mixed (Billig, Root, & Jesse, 2005; Deeley, 
2010; Frumkin et al, 2009; Hunter & Brisbin, 2000; Phelps & Dostilio, 
2008; Prentice & Robinson, 2007). 


Outcomes of Community-Based Research 

Often seen as a unique subspecies of service-learning, com¬ 
munity-based research (CBR) shares critical characteristics of 
service-learning, but also has special features that may influence 
student outcomes differently (Strand, Marullo, Cutforth, Stoecker, & 
Donohue, 2003). Most particularly, CBR tends to be discipline-spe¬ 
cific, and therefore has the potential for direct impact on a stu¬ 
dent’s perception of his or her academic major. Because of the dis¬ 
ciplinary nature of CBR, CBR is 
more likely to be delivered within 
the curriculum rather than as a 
cocurricular activity, since study 
design, data collection, analysis, 
and reporting are objectives 
commonly integrated into aca¬ 
demic courses (Strand et al, 2003). 

To date, CBR has not been 
studied nearly as extensively as 
has service-learning. A July 2010 
ERIC search of service-learning 
and outcomes yielded 384 results, 
while a search of community- 
based research and outcomes 
yielded six. As more CBR expe¬ 
riences and programs become 
integrated into college and uni¬ 
versity curricula, it becomes 
increasingly possible and important to identify features of program 
delivery (e.g., whether the course is a stand-alone CBR course or 


“As more CBR 
experiences and 
programs become 
integrated into 
college and university 
curricula, it becomes 
increasingly possible 
and important to 
identify...the extent 
to which [various] 
features affect student 
learning outcomes.” 
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CBR is a component of a regular course), how the CBR experiences 
are structured and delivered (e.g., whether reflection activities are 
built into the experiences, whether final products are produced, 
and whether final products, if produced, are shared in classes and/ 
or with community partners), and the extent to which these and 
other features affect student learning outcomes. 

Nearly all published discussions of CBR student outcomes 
are case studies (e.g., Willis, Peresie, Waldref, & Stockman, 2003; Puma, 
Bennett, Cutforth, Tombari, & Stein, 2009). Although the results are 
compelling, such studies make it difficult to generalize results 
beyond the specific experiences described. The authors found 
only one study that used a survey to assess CBR outcomes (Lewis & 
Niesenbaum, 2005), yet even this study conflates CBR and service¬ 
learning. In fact, the authors were unaware of a survey instrument 
that assesses CBR specifically along several dimensions of student 
learning outcomes familiar in the literature on service-learning, 
using conceptually valid and statistically reliable scales, and that 
can be implemented across institutions. 

The instrument reviewed in this article seeks to fill this gap. The 
authors believe that the CBR Student Learning Outcomes Survey 
has the potential to assess learning outcomes at student, course, 
and institutional levels, providing a common means of evaluating 
CBR that can focus research efforts across institutions and help 
identify specific strengths of CBR, including program features that 
enhance students’ experiences. 

Survey Development 

Identifying Potential Outcomes and Creating 
Constructs 

IRB approval was secured prior to the study. During 2007- 
2008, Cutforth visited six institutions with active CBR programs. 
He conducted over 30 individual and focus group interviews with 
undergraduate students who had experienced CBR. Altogether, 
over 70 students were interviewed. Respondents were undergrad¬ 
uate students from a wide range of majors, including the natural 
and physical sciences, social sciences, humanities, psychology, and 
business. The interviews allowed Cutforth to gain insights into the 
context in which the students’ CBR experience was taking place, 
including interactions in the classroom and community; how 
students encountered issues of race, class, gender, and other dif¬ 
ferences in their communities; and their recommendations for 
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improving the quality of CBR courses. Students provided varied 
and specific examples of their CBR experiences, and discussed and 
reflected upon the short- and long-term benefits they had expe¬ 
rienced, as well as challenges. Each discussion lasted from 30 to 
60 minutes. Interview questions focused on the extent to which 
students’ CBR experiences contributed to their personal, social, 
and cognitive development, as well as the extent to which their 
experiences influenced their thinking about future coursework and 
career choices. 

Each interview was digitally recorded, transcribed, and ana¬ 
lyzed. Using the constant comparison technique (Boeije, 2002), five 
themes were identified. The themes became constructs that con¬ 
stitute the framework of the CBR survey: development of academic 
skills, enhanced educational experience, increased civic engagement, 
development of professional skills, and personal growth. Each of 
these constructs is defined and discussed below. Table 1 summa¬ 
rizes the construct definitions. 


Table 1. CBR Learning Outcome Survey Constructs and Definitions 

Construct 

Definition 

Academic skills 

Cognitive skills related to academic learning 

Educational experience 

Affective outcomes that enhance the overall college expe¬ 
rience, including finding one’s passion, enhancing one’s 
interest in one’s major, and clarifying a career path 

Civic engagement 

Cognitive, affective, and behavioral outcomes related to 
community participation 

Professional skills 

Skills, behaviors, and attitudes that enhance efficacy in the 
workplace 

Personal growth 

Affective outcomes related to understanding oneself, 
including personal insights and transformation 


Academic Skills. Academic skills pertains to cognitive skills 
related to academic learning. Many student comments in the 
interview phase of the study highlighted the value of CBR in 
strengthening academic skills. Examples are 

• I remember more facts because it is something that 
you actually witness. 

• [Because of my CBR experience,] I know how to write 
an opening, a background section, a methodology, an 
analysis, and [a] conclusion. 
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• I revised my survey for my community partner about 
15 times so that has gotten me way ahead of working 
on my thesis survey. ... I know what works, what 
doesn’t, what people are hesitant to answer, how to 
phrase things. 


Educational Experience. Whereas academic skills focuses 
on cognitive outcomes related to coursework, educational expe¬ 
rience focuses on affective outcomes that enhance the overall 
college experience, including finding one’s passion, enhancing 
one’s interest in one’s major, and/or clarifying a career path. 
Several students commented on how CBR broadened and 
deepened their college experience: 

• Once the authors had the patterns and themes and 
fitted them together, I found that [research] was some¬ 
thing that I did enjoy. It made my mind happy. 

• Research is something that could interest me in a way 
that I had not thought of [because] of my narrow defi¬ 
nition of research. 

• I feel like you are doing research for a purpose. You 
are not just doing it for the sake of a grade or test; you 
are doing it because someone can actually use what 
you are doing. So it pushes you further to want to do 
the research. 

• Do I want to be in the field, hands on doing something; 
or do I want to be in the background doing research 
and that sort of thing? . . . [CBR] is feeling out what 
is right for you, what you can deal with and what you 
can’t. 

• CBR gives me an idea of the different things that I 
could do with my major, doing program evaluations, 
or research for people under a grant. 


Civic Engagement. Civic engagement is often touted as a 
benefit of service-learning and community-based research. In 
the CBR outcomes survey, civic engagement includes cognitive, 
affective, and behavioral outcomes related to community 
participation. Four items make up the civic engagement scale, 
which probes understanding those who are different from 
oneself, clarifying one’s values, and assessing one’s likelihood 
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of voting. Throughout our interviews, students commented 

frequently on the impact of CBR on their civic engagement. 

• CBR gave me a better sense of the community. We get 
so zoned into what is happening on campus and you 
kind of forget that you are in a larger city and there is 
life outside... 

• One of the hardest things was the sheer emotion of the 
things that I experienced. Seeing people in situations 
different from my own: the kids who are hungry or 
sick or have never been to the dentist.... 

• I feel that my background and how I look as the rich 
white person, my background is very privileged. When 
I look into the future, I would love to be working with 
a more diverse group and not stick out like a sore 
thumb and have to earn people’s trust. 

• CBR gave me a different perspective on people in gen¬ 
eral. It is hard to explain, but it changes you talking to 
people and seeing the difficulties that they face and 
how they have been able to overcome them or how 
they have maybe not been able to overcome them yet. 

• Sometimes you are a little close-minded and you put 
stereotypes on other people. But when I sat down 
and listened to [community members’] stories, I put 
myself in their shoes and realized that living in the 
city is completely different from where I grew up. You 
try not to stereotype someone who is 16 and pregnant. 

You try not to judge at all and listen and try and learn 
from what their experience was. 


Professional Skills. Professional skills refers to skills, 
behaviors, and dispositions that enhance efficacy in the 
workplace. Students described many activities related to 
their CBR experiences that they felt helped prepare them for 
professional careers. Skills probed include resolving conflicts, 
running meetings, delegating, listening to others, and working 
as part of a team. Comments pertaining to development of 
professional skills were pervasive across student interviews: 

• You can’t be shy. You have to be able to deal with 
people. 
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• To be on the spot and be able to think quickly and 
come up with ideas and have a conversation has been 
something that I am getting better at every time I do it. 

• You learn very quickly to prioritize. Sometimes you 
have to push the community partners more than they 
are pushing you. Even though it is their project, you 
need to stay on them, especially when you have dead¬ 
lines to meet. 

• There is a huge dif¬ 
ference when you are 
writing an email and you 
are trying to phrase it to 
make people like you. 

[But] you want them to 
do the work and actually 
tell them that they have 
an obligation to do it. 

So it is hard to find the 
middle ground. 

• I learned that you can 
rely on other people to 
get things done. 

• It makes you really focus 
on the fact that you have 
to work as a group to 
accomplish the goal. I 
would never have been 
able to come up with the 
survey the authors created without the help of all the 
group members. 

• CBR is learning how to work with people more effi¬ 
ciently, communicate better, which is definitely an 
important life skill and makes me a good candidate in 
the work field. 

Many students felt they had an edge in the job market because of 
their CBR experience. One student remarked: 

• In class, the professor will hold your hand a little or you 
can Google something. But [CBR] cannot be found on 
the Internet or in any textbook. You have to pick up a 
phone or you have to drive to that organization, you 


“In class, the professor 
will hold your hand a 
little or you can Google 
something. But [CBR] 
cannot be found on 
the Internet or in any 
textbook. You have to 
pick up a phone or you 
have to drive to that 
organization, you have 
to keep pursuing it until 
something becomes of 
it, because if you don’t 
do it, no one will do it.” 
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have to keep pursuing it until something becomes of 
it, because if you don’t do it, no one will do it. 


Personal Growth. Personal growth pertains to affective out¬ 
comes related to understanding oneself. Students spoke about 
significant internal transformations as a result of their CBR 
experiences. 

• CBR shaped my thought from, “Let’s work in a 
lab and make lots of money” to “Money is not the 
most important thing, so maybe your career can 
be important in a different way.” Being involved 
in the community helped me realize that I want to 
do something that helps other people, something 
rewarding, not necessarily money-wise but morals- 
wise. [CBR] challenges you in a way that nothing else 
on campus can: not volunteering, not research papers. 

This forces you out of your comfort zone and seeing 
that you can live up to the challenges. 

• I go to a great university where everyone is sheltered, 
but now seeing the community and the challenges and 
difficulties that they face means that I have to do some¬ 
thing great with my life and give something back to 
the community. 

• I grew up white, suburban, middle class. [Through 
my CBR project] I saw a different kind of life, people 
being exploited, people being oppressed, and it really 
changed my political outlook, my social outlook, what 
I fight for in my everyday life, and what I stand for. 

• The CBR experience made me question a lot of the 
things that I had been going along with for a very long 
time. 

Survey Pilot 

The authors developed a pilot survey, which was deployed 
online during spring semester 2009, to students at institutions par¬ 
ticipating in a consortium dedicated to deepening and expanding 
the practice of CBR. The pilot version of the survey included 95 
items and subitems in four sections and took approximately 15 
minutes to complete. The first section identified the ways students 
experienced CBR (as part of a CBR course, in a non-CBR course 
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with a CBR project attached, in an independent study, etc.). This 
section of the survey also asked students to identify activities they 
undertook within their CBR projects, such as collecting data, ana¬ 
lyzing data, reporting in class, or undertaking a reflection activity. 
Also included in this section was a series of nine items in which 
students rated their CBR experience as mostly positive, mostly 
negative, or mixed. 

The second section contained 30 items reflecting the five 
dimensions of CBR noted above: academic skills, educational 
experience, civic engagement, professional skills, and personal 
growth. To help confirm the validity of the constructs, they were 
also assessed in a different way by nine items that followed within 
the same section. 

The third section asked for students’ demographic informa¬ 
tion, including institution, ethnicity, socioeconomic status, and 
history of volunteer and civic activities. 

The last section included two open-ended items. One invited 
students to provide any other comments about their CBR experi¬ 
ences, and the other asked students to comment on their experi¬ 
ence taking the survey. These open-ended items informed subse¬ 
quent survey development. 

The pilot version deliberately contained more items and types 
of questions than would be included in the final version. In some 
cases a given question was phrased in multiple ways, in order to 
determine which version yielded the most statistically reliable 
response. Some items tapped different dimensions of a construct 
in order to explore which dimensions, ultimately, would be most 
explanatory. 

The survey was posted online from March 1 through June 
6, 2009. A total of 192 respondents completed several items, and 
approximately 166 completed all or nearly all items of the entire 
survey. 

Respondents were asked to identify their academic institution. 
Fifteen institutions were identified by a total of 170 respondents. 
Those institutions represented by more than two respondents 
included Bowdoin College, Cabrini College, Lafayette College, 
Macalester College, Princeton University, Rice University, Stetson 
University, University of Alaska-Anchorage, University of Notre 
Dame, Western Carolina University, and Whitman College. The 
authors believe this sample reflects a good range of academic insti¬ 
tution types, based on Carnegie Foundation classifications (see 
Carnegie Classification of Institutions of Higher Education, available at 
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http://www.carnepefoundation.org/dassifications) . Nevertheless, the 
sample is limited geographically (representing more institutions 
in the eastern United States), and is skewed toward institutions 
that have unusually strong CBR programs compared to academic 
institutions nationwide. 

The majority of respondents (51.5%) were seniors, followed 
by sophomores (20.1%), juniors (19.5%), and freshmen (4.7%). 
Women made up 74.5% of the sample. Caucasians made up 76.8% 
of the sample, followed by Asians at 11%, African Americans at 
6.1%, and Hispanics/Latinos at 5.5%, with less than 2% of respon¬ 
dents being Alaska Natives and American Indians. In addition, 
6% of respondents identified themselves as “Other.” (Respondents 
could self-identify as multiple races or ethnicities, so percentages 
total more than 100.) A proxy variable was created for socioeco¬ 
nomic status (see discussion in “Demographic Analyses,” below). 
The variable describes a normal curve, ranging from a low of 8 to a 
high of 26 (mean=19), indicating that the sample population, like 
college students generally, is skewed toward middle- and upper- 
middle socioeconomic status. 

In this sample, women and seniors are overrepresented, and 
Caucasians are slightly overrepresented. Broader sampling in the 
future might result in a different profile of outcomes. However, the 
authors believe that the distribution of responses on the pilot sur¬ 
vey’s demographic variables reflected sufficient representativeness 
and variability to conduct the item-level analyses that follow. 

Results 

In the first section of the survey, students reported the aca¬ 
demic activities they undertook during CBR. Crosstab analyses 
showed that several categories could be collapsed, because over 70 
percent of those who reported having experienced one type of CBR 
also experienced another. For example, 92% of those who reported 
that they had defined a problem/issue also reported researching a 
problem/issue. Given such overlap, the authors determined it was 
not necessary to subdivide these research activities (see Table 2). 

Consistent with findings in prior research (Conway et al, 
2009; Eyler and Giles, 1999), students’ responses in the focus group 
interviews highlighted the fact that integration of CBR activities into 
classes, including reflection activities, enhanced their experiences. 
Therefore, a survey item asked students to estimate the proportion 
of CBR courses that included some sort of reflection activity. This 
item correlated r = .405 (p < .01) with total CBR outcome score and 
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r = .338 (p < .01) with the combined (eight-item) CBR experience 
score, both of which are discussed below. These correlations suggest 
a moderate association between reflection activities and students’ 
perceived quality of CBR experience. 


Table 2. Frequency of CBR Activities Experienced by Respondents 

CBR activity 

% participating in this activity 

Research problem/issue 

76 

Define a problem/issue 

70 

Collect data 

70 

Analyze data 

62 

Report results orally in class 

58 

Attend meetings with partners 

42 

Implement project with partners 

21 

Report to policy-makers 

19 

Present at a conference 

16 

Report to partners 

5 

Other 

4 

Assessing CBR Overall Experience 

Predictably, students’ experiences with CBR will vary. Making 
sense of CBR outcomes requires accounting for students’ impres¬ 
sions of the overall quality of their CBR experiences. A series of five 
items probed various dimensions of students’ CBR experiences, 
including the extent to which 

U3K was miegrarea mio courses, 
supported by faculty, and appre¬ 
ciated by community partners; 
whether CBR activities were 
useful; and whether the student 
had voice in or control over the 
process. These items cover most 
of the best practices identified in 
the CBR literature (Puma et al, 

“Making sense of CBR 
outcomes requires 
accounting for students’ 
impressions of the 
overall quality of their 
CBR experiences.” 

2009; Stocking & Cutforth, 2006; 
Strand et al, 2003; Weinberg, 2003). 
Response options to these items 
No.” 

were “Mostly Yes” and “Mostly 
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The final item of this section, Overall CBR Experience, asked 
students whether their experience was positive overall, to which 
they could respond “Mostly Yes,” “Mostly No,” or “Mixed.” This 
single item was correlated with responses to the previous five items, 
which probed more specifically the quality of respondents’ CBR 
experiences. Table 3 shows inter-item correlations among CBR 
experience items as well as their correlation with the total learning 
outcome scores (see Learning Outcome Scales, below). 

Scores on the five CBR experience items correlated with the 
overall CBR experience item at r = .647 (p < .01). This is a strong 


Table 3. Correlations Between Overall CBR Experience Item, 

Combined CBR Experience Score, and Total Outcome Score 

CBR experience items 

Overall CBR 
experience 
item only 

Total CBR 
experience score 
(all 6 items) 

Total CBR outcome 
score (summed score 
of all five constructs) 

CBR projects were 
integrated into cource 
content. 

.216** 

.373** 

.186* 

Generally, 1 felt 
supported in my CBR 
experiences by college 
faculty/staff. 

.360** 

.581** 

.373** 

Interactions with 
community partners 
and community 
members were 
generally positive. 

.448** 

.731** 

455 ** 

My CBR activities were 
useful to my community 
partner. 

43 | ** 

.704** 

.489** 

1 have had some voice/ 

control over CBR 

activities I’ve been 

involved in. 

394 ** 

.642** 

.482** 

Overall, my CBR 
experiences have been 
positive. 

1.0 

.647** 

.520** 


*Correlation is significant at p < .05; **Correlation is significant at p < .01. 


correlation, suggesting that the five composite items largely (but 
not entirely) explain the result on the overall CBR experience 
item. These five summed items correlated r = .602 (p < .01) with 
the total CBR learning outcome score—the summed total of all 
items constituting the five learning outcome constructs. This, too, 
is a strong correlation. The correlation of the single, overall CBR 
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experience item with the total CBR learning outcome scores was 
r = .520 (p < .01). This is a moderately strong correlation, but is 
nevertheless impressive, considering that the correlation coefficient 
is depressed because overall CBR experience is only a single item 
and had only three response options (i.e., “Mostly Yes,” “Mostly 
No,” and “Mixed”). 

Learning Outcome Scales 

The survey pilot included 30 learning outcome-related items, 
each of which was on a 4-point scale: 1 = not at all; 2 = minimally; 3 
= moderately; 4 = extensively. Responses to these items were factor 
analyzed using principal components extraction with an Eigen 
value of 1 as the cutoff. This analysis was followed by a Varimax 
rotation. The principal components analysis revealed six factors 
that explained approximately 73% of response variance. 

The first factor corresponded to the five items of the profes¬ 
sional skills construct. This construct explained 58% of the overall 
variance, suggesting that this dimension of CBR is a critical benefit 
for many students. The second factor reflected four civic engage¬ 
ment items that explained 6.8% of response variance. Four items 
in the educational experience construct made up the third factor, 
which explained 6.5% of response variance. The fourth factor was 
academic skills. Three items loaded on this factor and explained 
4.5% of response variance. (Four items on the revised survey con¬ 
stitute this scale; three items were used in the pilot and a fourth was 
added when the current version was deployed). Personal growth 
was the fifth factor and explained 3.6% of response variance. A 
sixth factor explained 3.4% of response variance and was made up 
of two items pertaining to public speaking skill and confidence. 
The authors determined that this factor contributed minimally to 
overall results, and therefore it was dropped from the revised ver¬ 
sion of the survey. 

After removal of items that correlated very highly (r = .80 or 
higher) or that failed to have strong explanatory value (Eigen values 
less than 1.0), 19 items remained. Four experimental items were 
added in the current deployment. As a result of these analyses and 
revisions, estimated time to complete the survey dropped from 15 
minutes to 10 minutes. 

Scale Reliabilities 

The 19 items making up five constructs were analyzed for 
internal consistency using Cronbach’s alpha. Reliability of each 
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of the five factors and a factor created from a combined total are 
shown in Table 4. 


To summarize, 19 items can be summed to create a total CBR 
learning outcomes score that has extremely high reliability (a = 
0.95). The five factors that contribute to the overall CBR learning 


Table 4. Cronbach’s Alpha Reliabilities for Revised Factors 

Scale 

Number of items 

Cronbach’s alpha (a) 

Overall CBR outcomes 

19 

a = 0.95 

Professional skills 

S 

a = 0.91 

Civic engagement 

4 

a = 0.86 

Educational experience 

4 

a = 0.87 

Academic skills 

3 

a = 0.80 

Personal growth 

3 

a = 0.94 


outcome variable have reliabilities ranging from a = 0.80 to a = 
0.94. 

The authors created unweighted, scaled scores for each of the 
five constructs listed above and for the scale as a whole. Most of the 
composite scores were inter-correlated moderately, which is desir¬ 
able, since it suggests that each factor is assessing a different facet 
of an underlying phenomenon. As can be seen in Table 5, all scales 
correlate moderately or strongly with total CBR learning outcomes, 
indicating that each subscale captures an important aspect of stu¬ 
dents’ overall perceptions regarding the benefits of taking classes 
that include CBR. Several moderate correlations indicate that each 
scale is measuring something similar about CBR outcomes, but 
also something unique. This, combined with the high coefficient 
alphas previously reported, suggests that each scale can be used to 
create scaled scores for each of the five constructs that comprise 
the survey. 

Demographic Analyses 

The authors analyzed construct data to see whether there were 
differences based on gender, race, or socioeconomic status (SES). 
Using analyses of variance calculations (ANOVA), the authors 
detected no significant differences among any groups on total CBR 
learning outcomes, nor for any of the five subscale scores. As a 
result, the authors concluded that the five scales and the combined 
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total CBR learning outcome scale reflect CBR outcomes that are 
not biased based on students’ sex, race, or socioeconomic status. 

In this survey, the authors used three items to determine SES 
(Donaldson, Lichtenstein, & Sheppard, 2008). Two items are mother’s 
and father’s highest level of education, because they are generally 
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known by students and they have a good track record in the research 
literature as correlating with income. The authors combined those 
responses with respondents’ self-reported SES to come up with a 
single SES score. In combining the measures, they weighted mother’s 
and father’s education equally (if one was missing, they used the 
remaining score for both), combined them, and weighted the 


** Correlation is significant at the 0.01 level (2-tailed). 
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result equally with self-reported SES. In this sample, the combined 
score of mother’s and father’s education correlated r = .473 (p < 
.05) with student self-reported SES, suggesting that different but 
related information is obtained using the two measures together, 
rather than one alone. The authors believe that this approach yields 
a more accurate SES proxy than traditional measures, militates 
against research bias, and validates respondents’ perceptions of 
their own socioeconomic status. 

Summary and Further Research 

The goal of this research was to begin to codify student learning 
outcomes of CBR in order to assess best practices in CBR courses. 
To do this, the authors developed a survey instrument designed 
to validly and reliably assess student learning outcomes of CBR 
at the course, institutional, and national levels. The 19 items (plus 
four experimental items) that comprise the five CBR learning out¬ 
come constructs can be examined independently or summed into 
a combined scaled score. The constructs include academic skills, 
educational experience, civic engagement, professional skills, and 
personal growth. 

It is worth highlighting that the professional skills factor 
explained 58% of the total response variance in our pilot survey. 
This prominence in the development of organization and leader¬ 
ship skills is congruent with the results of other studies of service¬ 
learning (see Eyler & Giles 1999; Moely, Furco, & Reed, 2008). 

In addition to shortening and strengthening the CBR outcome 
scales, other revisions were made as a result of the pilot. The authors 
added three items to the CBR experience section based on student 
comments in the open-ended portion of the survey. The first item 
asks students to rate whether the term provided sufficient time to 
execute CBR projects. Respondent comments that prompted this 
addition include 

• What detracted most from the CBR experience was 
the time limitations in dealing with a community 
partner over the course of only a single semester.... 

• After the semester ended, our project community 
partners were still interested in receiving feedback 
and help from us, but the authors had moved on to 
different courses.... 
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A second item was added that asks respondents about the 
workload of CBR. This item was created based on comments such 
as the following: 

• [The] major problem I had with this project was that 
half of the project was scheduled outside of class and I 
had to miss another class to do this project—the week 
before finals. 

A third item was added asking students to rate whether their 
CBR projects, in general, were organized and expectations were 
clear. Several students offered qualitative comments in this regard, 
such as 

• A little unorganized, directions weren’t very clear. 

• I like CBR; however, some students may need more 
guidelines or frequent check-in with their instructor. 

Respondents of the pilot survey also commented that the 
instrument would be strengthened if students had the opportunity 
to explain the nature of their CBR projects. Because one objective 
of the survey was to identify a range of CBR practices nationwide, 
on the revised survey the authors added an open-ended item that 
allows respondents to briefly describe a CBR project. 

The revised survey is currently being used in a national 
study of CBR outcomes, conducted by Princeton University. The 
survey can be accessed at https://princetonsurvey.qualtrics.com/ 
SE?SID=SV lYUKLLiSOIsxLQE (note underscore between “V” 
and “1”). Any student from any college or university who has 
experienced CBR is invited to participate. The authors ask that fac¬ 
ulty make students aware of this 
link. The survey will be available 
through spring 2012 at the URL 
shown. Princeton will collect the 
data, perform the analyses, and 
report the results back to faculty 
members and institutions whose 
students participate. 

Furthermore, the survey 
may be used by educators as long 
as no monetary gain is associated 
with its use. The web version 
assesses CBR student learning 
outcomes for cumulative CBR experiences. The authors have also 
created a version that can be used to assess outcomes related to an 


“In addition, a 
revised version of this 
instrument could be 
used to study outcomes 
of CBR compared to 
traditional instruction 
or other pedagogies.” 
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individual course (see the Appendix). Interested parties may con¬ 
tact the authors regarding how to analyze the survey. 

The national study alluded to above seeks to confirm psycho¬ 
metrics of the current survey and to begin to codify outcomes 
of CBR that might be related to delivery types and program fea¬ 
tures. Over time, local and national norms could be established 
that would allow analyses of CBR outcomes by institution type, 
region, delivery type, class standing, or other demographic vari¬ 
ables. If scale reliabilities hold, the course-based version of the CBR 
Outcomes Survey could be used diagnostically. For example, if a 
CBR class had a t-score of 35 (one and a half standard deviations 
below the mean) for civic engagement, but scaled scores in the 
other areas above 50, this would suggest that more attention should 
be devoted to this aspect of the CBR experience the next time the 
class was offered. In addition, a revised version of this instrument 
could be used to study outcomes of CBR compared to traditional 
instruction or other pedagogies. 

The authors believe that the instrument they described in this 
article can help quantify outcomes of CBR and hope that this instru¬ 
ment will help proponents of CBR assess their efforts, better under¬ 
stand this dynamic pedagogy, and assist them in making improve¬ 
ments, ultimately heightening students’ learning experiences while 
conducting course-related research in authentic settings. 
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Appendix 


COMMUNITY-BASED RESEARCH 
COURSE SURVEY 

JtMve/ZOlO 

This survey is part of a national study on the outcomes of CBR. Your answers will be very important in 
helping colleges and universities design CBR programs. This survey will take less than 10 minutes to 
complete. We appreciate your honest responses to the questions below. 


> For each of the following sections, please reflect on A SINGLE CBR COURSE that you 
have taken. A CBR course might not have had CBR in the title but might have involved 
collecting data and/or conducting research for a class or community-based 
organization. 

> Research conducted in the community primarily for academic purposes DOES NOT 
COUNT. Research must have been in the service of a community partner. 

> Your responses are anonymous. 


I. About Your CBR Experiences 


la. Course Title:_ 

Department:_ Course Number:_ 

Instructor(s):_ 

Term:_ Institution:_ 

lb. Which description is most accurate for the above course? (Please check only one) 

□ CBR Course with project or internship 

□ CBR Theory Course, no project or internship 

□ Non-CBR course that included a project 

□ Independent CBR Project, Thesis, or Internship 

□ CBR Internship Only 

□ Other _ 

2. Please check all of the activities you have experienced in CBR courses: 

□ Researched a problem/issue 

□ Attended meetings with community partners 

□ Interacted with community members and/or partners outside of meetings 

□ Participated in a community-based program/project 

□ Reported CBR findings in class (orally, in writing, or via technological media) 

□ Reported CBR findings to community partners (orally, in writing, or via technological media) 

□ Reported CBR findings to policy-makers (orally, in writing, or via technological media) 

□ Presented CBR findings at a conference 

□ Other:_ 
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3. Did the above course involve some sort of personal reflection activity (e.g., reflective 

paper, meeting with instructor, journal-keeping)? Yes_ No_ 

4. Please choose the answer that best fits your experience in this course. NA=Not Applicable. 


a. There was sufficient time in the term to execute my 
CBR project. 

□ Mostly yes 

□ Mostly no 


b. CBR project tasks and/or expectations were clear. 

□ Mostly yes 

□ Mostly no 


c. The amount of time taken for the course was realistic 
for the credits 1 earned. 

□ Mostly yes 

□ Mostly no 

□ NA 

d. The CBR project was integrated into course content. 

□ Mostly yes 

□ Mostly no 

□ NA 

e. Generally, 1 felt supported in my CBR experience 
by college faculty/staff. 

□ Mostly yes 

□ Mostly no 

□ NA 

f. Interactions with community partners and community 
members were generally positive. 

□ Mostly yes 

□ Mostly no 

□ NA 

g. My CBR activities were useful to my community partner. IHiviostly yes 

□ Don't Know 

□ Mostly no 

h. 1 had some voice/control over the CBR activities in which 1 

was involved. OMostlyyes 

□ Mostly No 

□ NA 

i. Overall, my CBR experience in this course was positive. 

□ Mostly yes 

□ Mixed □ Mostly no 


5. In the space below, please briefly describe your CBR project. 
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CBR Learning Outcomes Survey 
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II. CBR OUTCOMES 



My participation in this CBR experience: 

Not at 

All 

Minimally 

Moderately 

Extensively 

a. 

Strengthened my analytical skills 





b. 

Improved my academic writing skills 





c. 

Improved my research skills 





d. 

Enhanced my understanding of academic content 





e. 

Enhanced my understanding of local issues 





f. 

Enhanced my understanding of social issues 





g- 

Deepened my understanding of others who are not 
like me 





h. 

Helped me empathize with those who have 
different racial or religious backgrounds than 1. 





1 

Enhanced the likelihood that 1 will participate in 
civic activities. 





j- 

Enhanced the likelihood that 1 will vote. 





k. 

Helped clarify my values 





1 . 

Increased my interactions with faculty. 





m. 

Increased my interest in my major. 





n. 

Improved my interest in college. 





0 . 

Clarified my career path. 





P- 

Improved my skills with conflict resolution. 





q- 

Improved my ability to run meetings. 





r. 

Improved my ability to delegate. 





s. 

Improved my ability to listen to others. 





t. 

Improved my ability to work as part of a team. 





u. 

Helped improve my personal qualities. 





V. 

Improved my ability to consider others' 
perspectives. 





w. 

Deepened my understanding of myself. 






Please share any other thoughts about your CBR experience and how it has impacted you. 
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III. About You 


This information is most important for helping us understand your responses. 
We greatly appreciate your responses to the following questions. 


WHAT’S YOUR ACADEMIC INSTITUTION? 


YOUR CLASS STANDING IS: DFreshman DSophomore □ Junior DSenior IHMaster's/Doctora! 

YOUR SEX: □ Female □ Male 


RACE/ETHNICITY (Check all that apply): 

□ African American 

□ American Indian or Alaskan Native 
Islander 

□ Asian American 


□ Caucasian □ Other 

□ Hispanic or Latino/a 

□ Native Hawaiian or Pacific Islander 


MOTHER'S HIGHEST LEVEL OF EDUCATION: 

□ N/A or Don't Know 

□ Less than High School 

□ Completed High School or GED 

□ Some college, but no degree 

□ 2-Year/Technical/Vocational Degree 

□ 4-year/Bachelor's Degree (B.A./B.S., etc.) 

□ Master's Degree (M.A., M.S., etc) 

□ Professional Degree (Ph.D., M.D., J.D.) 


FATHER'S HIGHEST LEVEL OF EDUCATION: 

□ N/A or Don't Know 

□ Less than High School 

□ Completed High School or GED 

□ Some college, but no degree 

□ 2-Year/Technical/Vocational Degree 

□ 4-year/Bachelor's Degree (B.A./B.S.,etc.) 

□ Master's Degree (M.A., M.S., etc.) 

□ Professional Degree (Ph.D., M.D., J.D.) 


THINKING BACK ON WHEN YOU WERE GROWING UP, WOULD YOU DESCRIBE YOUR FAMILY'S 
INCOME AS...? 

□ Low □ Lower-Middle DMiddle DUpper-Middle DHigh 


© THANKS FOR YOUR COOPERATION! © 
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